Generic signal fusion framework for multi-modal localization

ABSTRACT

The present invention provides a signal fusion system that is a probabilistic system fusing an arbitrary combination of heterogeneous signals for determining the location of electronic devices. The system includes a signal sampling device for detecting one or more signals emitted by an electronic device to be located, including geolocation signals, WiFi signals, Bluetooth signals, 4G communication signals, 5G communication signals, geomagnetism signals, or inertial navigation system signals (INS). A likelihood processor cooperates with the signal sampling device to receive information about selected sampled signals, and creates a grid of reference points for an interested area in which the electronic device may be located. The likelihood processor independently computes, for each selected sampled signal, a location likelihood that is a probability of observing the sampled signal given that the electronic device is located at different reference points in the grid.

FIELD OF THE INVENTION

The present invention generally relates to localization techniques fordetermining the position of electronic devices/users of electronicdevices. More particularly, the present invention relates to aprobabilistic system fusing an arbitrary combination of heterogeneoussignals for determining the location of electronic devices.

BACKGROUND OF THE INVENTION

Localization technology has wide range of applications in navigation,location-based marketing, geo-fencing, etc. For open outdoorenvironments, GPS usually provides an acceptable solution. However, inurban-canyon, semi-indoor or deep indoor settings, GPS accuracydeteriorates due to weak or unavailable signals. For complex indoorcases, fingerprinting emerges as a promising technique for localization.In fingerprinting, a site is first surveyed to label detected signalvalues with their locations, the so-called “fingerprints.” Given thefingerprints, a user location can be estimated based on the signals thatare sampled. Various fingerprinting signals have been considered, suchas WiFi, Bluetooth, magnetic field-based signals, etc. Each signalcategory has its own strengths and limitations. For example, GPS worksreasonably well outdoors but may be unavailable indoors. Radio-frequency(RF) signals such as WiFi or Bluetooth are pervasive and differentiableover a long range, leading to its deployability in an indoorenvironment. However, due to multi-path and fading effects, RF signalsgenerally suffer from relatively high noise. The localization accuracyalso depends considerably on the strength and density of the signals.For some signals such as WiFi, their sampling rate in mobile devicessuch as phones may be infrequent (once every tens of seconds), whichadversely affects user experience. Geomagnetism is another signalcategory that has been explored. Its strengths are its omnipresence (noadditional infrastructure), fast sampling rate (tens of samples persecond) and low noise. Its drawback is global ambiguity where a certaingeomagnetic sequence may be matched to multiple places in the area ofinterest for localization.

Inertial sensors are available in almost every mobile device, and may beused to provide user movement information. However, inertial sensorinformation only provides relative location information and errors mayaccumulate and diverge over time. As a result, INS is often combinedwith other signals to enhance localization accuracy.

In view of the complementarity of the various signals used forlocalization, recent research has been focusing on fusing differentsignals to combine their strengths while mitigating their weaknesses.However, fusion techniques have considered only two or three signalscombined in a highly specialized and customized manner according to thespecific signal characteristics. As a result, extending the fusiontechniques to include other signals is typically not possible. Whilesignals for a given technique are often assumed to be fully available atthe time of a localization decision, in reality, due to widely differentsignal sampling rates, missing signal values occur. Attempts tocompensate for missing signal values include using the last measuredsignal values, making predictions based on history, or reducing thelocalization frequency to the slowest sampling rate. None of thesecompensation techniques are satisfactory. Thus, there is a need in theart for a generic signal fusion framework for multi-modal localizationsystems and methods.

SUMMARY OF THE INVENTION

The present invention overcomes the problems of prior art systems bysupporting signal addition and removal at any time. It is a platformincrementally extensible to new signals without the need for retrainingthe whole system.

The present invention provides a signal fusion system and method termed“SiFu.” The SiFu system is a probabilistic system fusing an arbitrarycombination of heterogeneous signals for determining the location ofelectronic devices. The system includes a signal sampling device fordetecting one or more signals emitted by an electronic device to belocated, including geolocation signals, WiFi signals, Bluetooth signals,4G communication signals, 5G communication signals, geomagnetismsignals, or inertial navigation system signals (INS). The signalsampling device uses heterogeneous sampling rates for each of theselected signals.

A likelihood processor cooperates with the signal sampling device toreceive information about selected sampled signals, and creates a gridof reference points for an interested area in which the electronicdevice may be located. The likelihood processor independently computes,for each selected sampled signal, a location likelihood that is aprobability of observing the sampled signal given that the electronicdevice is located at different reference points in the grid. Thelocation likelihoods are combined in a weighted fashion in a fusionmodule.

A particle filter uses the location likelihood to update a particleweight in the particle filter to determine the location of theelectronic device.

The likelihood processor may include various modules for determining thelikelihoods of different types of signals. For example, a neural networkmodule may be used for RSSI vectors (e.g., Bluetooth, WiFi). A Bayesiananalysis module may be used for determining the likelihood ofgeolocation signals. A dynamic time warping module may be used for theanalysis of sequence-based signals (magnetic field-based signals).

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the present disclosure are readily understood from thefollowing detailed description when read with the accompanying figures.

FIGS. 1A and 1B schematically depict the SiFu system and method of thepresent disclosure;

FIG. 2 is a schematic view of a denoising autoencoder;

FIG. 3 is a schematic view of a dual neural network configuration;

FIG. 4 is a plot of the mean error of different schemes for spot-basedRSSI vectors in different locations;

FIG. 5 is a plot of the mean error of different localization schemes atdifferent sites;

FIGS. 6A-6C show the cumulative distribution of localization errors atdifferent sites;

FIGS. 7A-7C show real-time localization errors at different sites;

FIGS. 8A-8C show mean localization error at different sites withdifferent sizes of computation grids;

FIG. 9 depicts computation time with different grid sizes;

FIG. 10 depicts mean localization error at different sites withdifferent magnetic field matching ranges;

FIG. 11 depicts computation time with different magnetic field matchingranges;

FIG. 12 shows mean localization error at different sites with differentσ_(m).

FIG. 13 shows mean localization error at different sites with differentσ_(w).

FIG. 14 depicts the mean error of different localization schemes atdifferent sites with sparse WiFi sampling;

FIG. 15A-15C shows the cumulative distribution of localization errors atdifferent sites with sparse WiFi sampling.

DETAILED DESCRIPTION

Turning to the drawings in detail, FIG. 1A illustrates a signal fusion(SiFu) system. The signal fusion system is termed “SiFu.” The SiFusystem is a probabilistic system fusing an arbitrary combination ofheterogeneous signals for determining the location of electronicdevices. As seen in FIG. 1A the system 10 includes a signal samplingdevice 20 for detecting one or more signals 60 emanating from anelectronic device 50 to be located, including geolocation signals, WiFisignals, Bluetooth signals, 4G communication signals, 5G communicationsignals, geomagnetism signals, or inertial navigation system signals(INS). The signal sampling device uses heterogeneous sampling rates foreach of the selected signals 60. The signal sampling device may beselected from receivers, transceivers, mobile phones, smart watches andIOT devices.

A likelihood processor 30 cooperates with the signal sampling device 20to receive information about selected sampled signals, and creates agrid of reference points for an interested area in which the electronicdevice 50 may be located. The likelihood processor independentlycomputes, for each selected sampled signal, a location likelihood thatis a probability of observing the sampled signal given that theelectronic device 50 is located at different reference points in thegrid.

The likelihood processor may include various modules for determining thelikelihoods of different types of signals. For example, a neural networkmodule 34 may be used for RSSI vectors (e.g., Bluetooth, WiFi). ABayesian analysis module 32 may be used for determining the likelihoodof geolocation signals. A dynamic time warping module 36 may be used forthe analysis of sequence-based signals (magnetic field-based signals).

The independently computed likelihoods are fused in a weighted fashionin a likelihood fusion module 40, which assigns weights according to theindividual model performance at each grid point in the training processof the fingerprints.

A particle filter 70 uses the location likelihood to update a particleweight in the particle filter 70 to determine the location of theelectronic device.

SiFu adopts a probabilistic framework as illustrated in FIG. 1B. Itconsiders various classes of signals depending on how they are processedfor localization. For example, sequence-based signals base localizationon a group of consecutive signal samples (e.g., magnetic field), andspot-based signals base localization on a single sample. Spot-basedsignals are further sub-divided into a first group in which the signalinput is already in the form of global latitude and longitudecoordinates (e.g., GPS or Beidou), and a second group which use a vectorof received signal strength indicators (RSSIs). The second groupincludes signals such as WiFi and Bluetooth.

In operation, the signal sampling device 20 samples one or more signalsemitted from the electronic device to be located 50. A sampled signal isfed into a likelihood processor 30 from the signal sampling device,which independently converts the signal value into a location likelihoodon pre-defined grids in a feasible area of a map relating to a potentialdevice location. The Bayesian module 32 converts the global coordinatesand accuracy to a likelihood on the pre-defined grids. For a signalstrength vector, the neural network module 34 returns the likelihood byvector comparison using a neural network, discussed in more detailbelow. For sequence-based signals, the dynamic time warping module 36estimates the likelihood through matching the input sequence againstfingerprints on the grid points. Note that the likelihood computationmodules 32, 34, and 36 are generic, i.e., they are applicable todifferent signal types within a class. As such, the three modules arecapable of dynamically analyzing various groups of signals, making thesystem capable of adding or dropping a category of signals to beanalyzed based on current conditions. Normally, a signal is dynamicallydropped when it is not observed at the localization time. This canhappen because signal sampling rates are different for different signalsand also due to limited signal coverage. Similarly, a signal may bedynamically added when it is observed again. The ability to dynamicallyadd and drop signals is a considerable strength of the presentinvention, making it able to adapt to different signal conditions andadapt to different environments/surroundings.

The independently computed likelihoods are fused in a weighted fashionin the likelihood fusion module 40. The weighted likelihood is thenintegrated with user movement information (as obtained from INS) usingthe particle filter 70, with feasible areas of the map as a constraint.

Because the SiFu system uses a probabilistic framework fusing anarbitrary combination of heterogeneous signals the system 10accommodates the presence or absence of signals due to different signalsampling rates, signal addition, signal removal, missed samples, etc.Therefore, system 10 achieves high elasticity and scalability in signalcombination without the need for retraining and redesigning the systemdue to the changing signal environment.

RSSI vector comparison typically cannot cope with missing ornon-overlapped signal values between two vectors seriously impactinglocalization accuracy. In the neural network module 34, a novel machinelearning technique is used which employs a denoising autoencoder tolearn the latent representation of signals in the area of interest forlocalization determination. In contrast to conventional approaches basedon raw signal readings (e.g., cosine similarity or Euclidean distance),the SiFu system 10 estimates likelihood in the latent space. Because thedeep features are learned from all fingerprint signals, the overcomesmissing values problem is overcome, achieving higher robustness andaccuracy.

Different signals may yield different localization accuracies atdifferent locations in the interested area. The poor performance ofsystem components may undermine the performance of the whole fusionlocalization system. Prior art approaches typically are customized forspecific signals and hence, cannot add new signals into the system ordrop previously-used signals. In the present invention, a weightedlikelihood in Bayesian analysis is used to fuse multi-modal signals.Through a location-dependent weighting for each signal, signals areintelligently combined to achieve high accuracy.

Operation of the Likelihood Processor and its Modules

The area of interest is discretized into grids of reference points (RP).A localization likelihood, that is, the probability that a device to belocalized is present at a reference point, is computed for each RP inthe grid. Note that for signals using a fingerprinting approach, afingerprint signal should be obtained at each RP. The symbols used inthe following equations are defined below in Table I:

TABLE I Major symbols in SiFu. Notation Definition S A vector containingall observed signals s An observation of a particular signal type x Thestates in the interested area d Dimension of WiFi latent representationL Number of layers in WiFi encoder/decoder σ_(w) The observation modelparameter σ for WiFi σ_(m) The observation model parameter σ formagnetic field r Magnetic matching range w_(ij) The weight at RP i forsignal j N Number of particles in particle filter

Mathematically, the probability that the device is positioned atdifferent locations given the observed signals, is denoted as p(x S),where x denotes the states (locations) in the interested area and S=(s₁,. . . , s_(n))^(T) is a vector containing n observed signals.

The particle filter, which will be discussed in further detail below, isused to estimate this distribution. As such, the measurement likelihoodp(S x) is determined for updating the weights of particles in theparticle filter. Since different signals are sampled by differentsensors, conditional independence among different signals is assumed.

The likelihood p(S|x) is given by

$\begin{matrix}{{p\left( {S❘x} \right)} = {{p\left( {s_{1},\ldots\mspace{14mu},{s_{n}❘x}} \right)} = {{n_{i = 1}^{n}{{p\left( {s_{i}❘x} \right)} \cdot p}\left( S \middle| x \right)} = {{p\left( {s_{1},\ldots\mspace{14mu},\left. s_{n} \middle| x \right.} \right)} = {\prod\limits_{i = 1}^{n}{p\left( s_{i} \middle| x \right)}}}}}} & (1)\end{matrix}$

For every signal s_(i), a measurement likelihood p(s_(i)|x) isindependently computed which is the probability of observing the sampledsignal s_(i) given the device is located at different RPs. This servesas the foundation for the generic fusion localization framework.

Geolocation data nay provide the global coordinates of the device in theformat of [latitude, longitude, accuracy]. GPS and Google Fused LocationAPI are two examples of this class of signal. While GPS is onlyavailable outdoors, Google API can locate devices virtually everywherethere is an Internet connection. In the two examples mentioned above,the reported accuracy is defined as the radius of 68% confidence,suggesting that there is a 68% probability that the true location isinside the circle centered at the reported location with radius equal tothe reported accuracy. Also, note that alignment between map coordinatesystems may be required.

This class of signal is modeled with a bivariate normal distributioncharacterized by a mean μ and covariance Σ. The mean vector isconstructed using the reported location as μ=({tilde over (x)}, {tildeover (y)})^(T), where ({tilde over (x)}, {tilde over (y)}) are thereported location in the local map coordinate system. The covariancematrix Σ of the distribution is approximated from the reported accuracy.With no prior knowledge, it is reasonable to assume that both dimensionsof the Gaussian distribution are uncorrelated and have the samevariance. Note that the (1 α) interval for a multivariate Gaussiandistribution is given by:

(x−μ)^(T)Σ⁻¹(x−μ)≤;χ_(p) ²(α)  Equation (2)

where μ and Σ are the mean vector and covariance matrix of the Gaussiandistribution χ_(p) ²(α) is the α_(p) quantile of χ_(p) ².

As(x−μ) represents the difference between a point x and the mean μ,using the localization accuracy, we can obtain

${\sigma_{x} = {\sigma_{y} = \frac{a}{\sqrt{2.27}}}},$

where a is the accuracy reported. Hence, the covariance matrix

$\Sigma = \begin{bmatrix}\sigma_{x} & 0 \\0 & \sigma_{y}\end{bmatrix}$

Then the measurement likelihood p(s x) is defined as given that locatingat x, the probability of the mean being equal to the reported location.By Bayes' theorem:

$\begin{matrix}{{p\left( s \middle| x \right)} = \frac{{p(s)}{p\left( x \middle| s \right)}}{p(x)}} & {{Equation}\mspace{20mu}(3)}\end{matrix}$

A uniform prior for the mean is placed as it is equally probableeverywhere, i.e., p(s) is constant. As p(x s) is the density function ofGaussian distribution, which can be computed with a known mean andcovariance matrix, the likelihood is given by:

$\begin{matrix}{{{p\left( s \middle| x \right)} \propto {p\left( x \middle| s \right)}} = {\frac{1}{2\pi\sqrt{\Sigma }}{\exp\left( {{- \frac{1}{2}}\left( {x - \mu} \right)^{T}{\Sigma^{- 1}\left( {x - \mu} \right)}} \right)}}} & {{Equation}\mspace{14mu}(4)}\end{matrix}$

RSSI Vector of Spot-Based Signals

For spot-based RSSI vectors, the likelihood computation at RP x_(i) inthe localization step obtains a similarity measure δ(s, t) between thesampled signal s and the fingerprint signal t at RP x_(i). By assuming anormally-distributed observation model on the similarity measure, theprobability of observing the RSSI vector s at the RP x_(i) is obtainedas:

$\begin{matrix}{{p\left( s \middle| x_{i} \right)} = {\frac{1}{\sqrt{\left( {2\pi\sigma^{2}} \right)}}e^{- \frac{\delta{({s,t})}}{2\sigma^{2}}}}} & {{Equation}\mspace{14mu}(5)}\end{matrix}$

where σ is the standard deviation of the normal distribution, which is aparameter for the sensitivity of the observation model.

The similarity measure can sometimes be obtained by comparing rawreadings. However, it is not good enough for signals of higher dimensionwhere signal noise and missing features may severely mislead thesimilarity computation, and eventually the likelihood computation.Therefore, it is proposed to make use of the latent representation ofsignals to compute a similarity measure. The intuition is that thelatent representation is the deep features learnt from all collectedfingerprint signals in the interested area. Signal noise and missingmeasurements will be implicitly considered in the latent representationgeneration. Using such features is expected to give a more precise androbust similarity measure. The comparison between two RSSI vectors on arepresentation with a fixed dimension is also fairer. For localizationpurpose, it is expected that if the physical locations of two signalsare close, their latent representations will also be similar.

Machine learning techniques are used to learn the signal latentrepresentation. First, a denoising autoencoder is trained. FIG. 2schematically depicts the structure of the autoencoder. The input is anoisy version of the fingerprint signals collected in the offline surveyphase. Noise can be injected in the following two ways: (1) randomlymasking features, which makes the model robust to the situation whensome features are missing, and (2) adding Gaussian noise to simulatelocal measurement errors. The autoencoder includes an encoder thatlearns to encode the noisy signal into a latent representation ofdimension d, as well as a decoder that recovers the original cleansignal from the representation. In the autoencoder of FIG. 2, bothencoders and decoders are made up of L fully connected layers.

Only one autoencoder is needed to learn from all fingerprints in theinterested region, instead of one autoencoder specifically for one RP.Consequently, the method of the invention requires less effort regardingdata collection and model training. Further, through learning to encodeand decode the signals, similar signals in the whole localization regionwill generate similar latent representations, making it useful forsignal differentiation for localization.

With the denoising autoencoder, we use a Siamese/dual network depictedin FIG. 3 to compute the similarity measure. In this architecture, twonetworks share the same weight W on two different inputs X₁ and X₂.Given the sampled signal S and fingerprint signal t, the similaritymeasure is computed as:

δ(s,t)=∥G _(w)(s)−G _(w)(t)∥  Equation (6)

where G_(w) denotes the neural network with weight W. Here, the networksare essentially the encoder part of the autoencoder and the similaritymeasure computes the Euclidean distance between the latentrepresentations of two RSSI vectors. A WiFi signal is taken as anexample for this category of signals. WiFi RSSI readings are widely usedin many fingerprint-based indoor localization systems. Because of signalattenuation during propagation, RSSI from each AP reflects how far thesignal propagates in physical space from that AP. A WiFi RSSI vector,which consists of RSSI values from all detected APs at the moment,contains information rich enough to give a rough estimation of thedevice location. However, due to multipath and fading effects, WiFisignals are noisy in indoor environment and sometimes a few APs may notbe detected. The denoising autoencoder addresses these problems andhelps generate a better similarity measure.

Sequence-Based Signals

Due to differences in the signal sampling device sampling rate anddevice movement (for example, a user carrying a device having a certainwalking speed), two sequences may not be aligned, making comparing twotemporal sequences very difficult. To mitigate this issue, a dynamictime warping (DTW) algorithm is used which can stretch or compress thetime dimension of sequences for optimal matching. The DTW algorithmfinds an optimal warping path for two sequences S and t. In the warppath, data points in sequence s are associated with data points insequence t. Finally, it returns the distance of the warp path as the sumof difference between data points in the warp path as:

$\begin{matrix}{{\delta\left( {s,t} \right)} = {\sum\limits_{k = 1}^{n}{{s_{ik} - t_{jk}}}}} & {{Equation}\mspace{14mu}(7)}\end{matrix}$

where sik and sjk are the k th data points on the warp path from thesequences s and t respectively.

The distance itself is a similarity measure, thus the likelihood isobtained based on a normally-distributed observation model as:

$\begin{matrix}{{p\left( s \middle| x_{i} \right)} = {\frac{1}{\sqrt{\left( {2\pi\sigma^{2}} \right)}}e^{- \frac{\delta{({s,t})}}{2\sigma^{2}}}}} & {{Equation}\mspace{11mu}(8)}\end{matrix}$

Since a geomagnetic field reading is only a 3-dimensional vector, itusually cannot provide sufficient information for localization. Forexample, ambiguity is common where the magnetic field values at two verydifferent locations are similar and this cannot be resolved easily.Hence, it is often necessary to consider multiple magnetic fieldreadings within a period of time to predict device location. Therefore,magnetic field is selected as the representative of sequence-basedsignals.

As the complexity of the DTW algorithm is O(MN) where M and N are thelengths of the sequences respectively, the efficiency of the SiFu systemis further improved by using fast DTW, a fast approximation of DTW thathas linear time complexity. In addition, the last user location is usedto limit the search space to speed up the matching process. The magneticmatching range denoted as r determines how large the magnetic matchingspace will be. Only the magnetic field fingerprints within r m from thelast predicted location will be matched against the sampled signal forlikelihood computation.

Likelihood Fusion

The independently computed likelihoods are combined into a jointlikelihood in order to update the particle weight in the particlefilter. Normally, this can be done using Eq. (1). Given the likelihoods,it only computes their product directly. However, these likelihoods maybe inaccurate because of the model defects, producing an unreliablefinal result. To further address the vulnerability of the individualmodels, the present invention uses a weighted likelihood. In general,weighted likelihood, L^(w) ^(ij) , is an approximation of Bayesianinference L. It is believed that weighted likelihood provides a goodapproximation to the posterior distribution with properly selectedweights. In localization, signals may suffer from ambiguous readings andnoise, thus it is possible to receive similar readings at two locationswith far physical distance. Such signals can seriously affect the modeltraining phase and consequently affect the performance of the model andthe shape of likelihood function.

In the present invention, the weighted likelihood is adopted to avoidany negative influences due to poor model performance by spatiallycontrolling the contribution of the signal to the likelihood. With asmall weight, the signal will contribute less since low likelihood willbe pushed closer to the high likelihood end. Discrimination betweensignals will be reduced. In the extreme case where the weight is 0, theweighted likelihood will become 1 no matter what the original likelihoodis. In this case, the contribution of the signal is removed and thedecision power is shifted to the other signals. In contrast, a signalwith a high weight will have likelihood values useful fordifferentiating the location.

In this weighted scheme, weights, which indicate the wellness of themodels, play an important role. Therefore, selection of weights isimportant for SiFu. Intuitively, signals with more distinguishablefeatures should be assigned a higher weight, and vice versa. In SiFu,weights are inferred from the performance of models employing afingerprint approach over all the area of interest. For signals that donot use a fingerprint approach, the invention assigns a weight of 1 sothat the original likelihood function is preserved.

In SiFu, weights are location-specific because signal distinguishabilitymay vary within the area of interest. Let w_(ij) be the weight of signaltype j at RP xi and Vij be the validation dataset consisting of thesignals of type j collected there. To find w_(ij) the localizationperformance is tested using a single signal source signal j on thevalidation dataset Vij. The localization error is computed as theEuclidean distance between ground truth and the predicted location. Theweight is then computed through:

w _(ij)=1−{tilde over (D)} _(i),

w _(ij)=1−{tilde over (D)} _(l)  (9)

where {tilde over (D)}_(l) is normalized mean error at RP x_(i).

The measurement likelihood of interest is approximated by:

$\begin{matrix}{{p\left( {s_{1},\ldots\mspace{14mu},{s_{m}❘x_{i}}} \right)} = {{n_{j = 1}^{m}{{p\left( {s_{j}❘x_{i}} \right)}^{w_{ij}} \cdot {p\left( {s_{1},\ldots\mspace{14mu},\left. s_{m} \middle| x_{i} \right.} \right)}}} = {\prod\limits_{j = 1}^{m}{p\left( s_{j} \middle| x_{i} \right)}^{w_{ij}}}}} & (10)\end{matrix}$

Particle Filter

Upon obtaining the weighted likelihood, the particle filter 70 is usedto combine the weighted likelihood with device (user) movement andinformation obtained from INS. Orientation and travel distance of thedevice are estimated from the readings of the gyroscope, magnetometerand accelerometer. A typical particle filter performs the following: 1)particle prediction, 2) weight update 3) location estimation, and 4)resampling.

The particle prediction step relies on the estimated orientation andtravel distance information to propagate the particle. In the weightupdate stage, the computed weighted likelihood is used. Likelihoods arecomputed at the RPs in the computation grid. While particles do notnecessarily fall exactly on the RPs, each particle will take thelikelihood value of its closest RP. Furthermore, if the particleviolates the map constraints such as moving across a wall, the weight isset to 0. The weight is then normalized.

Given a set of N particles ((x_(i), y_(i)), w_(i)), the device locationestimation step computes a device location by the weighted average ofthe positions of the particles as:

Equation 11

Finally, the resampling step corrects the set of samples based on theevidence. While more particles will be resampled in a region that has ahigher probability density, some diversity is also added to the samples.

Example

SiFu may be implemented in various hardware including computers equippedwith signal sampling devices, mobile phones, tablets, or custom hardwarethat includes the signal sampling device, likelihood processor, andparticle filter. The following experiment uses Android mobile phones toimplement the SiFu system. For this example, four signals of widelydifferent characteristics were selected: WiFi, magnetic field, GPS, andGoogle Fused Location API (which includes GPS, if any). Using thesesignals as examples, the example demonstrates their application in theinventive framework for multi-modal localization. However, the presentinvention is not limited to these signals, and may be equally and simplyextended to other signals. The example was performed in three areas, anindoor corridor, an indoor open-space area and a semi-indoor atrium areain a university campus. The example demonstrates that SiFu is highlyaccurate in localization, improving substantially the accuracy ascompared with other localization techniques (by more than 30%). Bydemonstrating an indoor and outdoor transition, it is proven that theSiFu system can manage an arbitrary combination of signals to achieveseamless roaming without switching algorithms between outdoors andindoors.

There are 91 APs in the corridor area covering a few long corridors, 93APs in the 6010 m² indoor open space area and 103 APs in the 6824 m²atrium area.

In the atrium area, the ability of the system to handle arbitrarycombinations of signals is tested. In some regions GPS is available andsome regions are WiFi and magnetic fingerprint regions. In theexperiment, the device moves from fingerprint region to non-fingerprintregion such that the set of signals used switches from WiFi, magneticand Google API to GPS and Google API. An In/Out Region Detection moduleis trained to detect whether a WiFi signal is observed within thefingerprint region in the online phase. If so, it is used it forlocation estimation. Otherwise, it is discarded.

The parameter settings in the Example are shown in Table II. Thesampling frequency is around one sample every few seconds for WiFisignals and tens of samples every second for magnetic field and otherINS sensors. Furthermore, Google Fused Location API returns a locationevery few seconds while the sampling rate of GPS is generally faster.

TABLE II Default parameters. Parameter Default value Grid size    1.5m N 1,000 d    32 L    3 r    9m σ_(w)    5 σ_(m) 10,000

In the Example, the present invention is compared to two prior artlocalization algorithms and Google API as well as a naive version of thepresent system.

-   -   WiDeep: This localization scheme applies stacked denoising        autoencoders to denoise WiFi RSSI. For every reference point, an        autoencoder is trained correspondingly. User location is        estimated by selecting the point whose autoencoder recovers a        WiFi RSSI reading the best. Noisy training readings are        simulated by noise injector which assumes Gaussian distribution        on environment noise.    -   F-Score-Weighted (FSW): This localization scheme measures weight        of WiFi and magnetic field through F-score. Location is        estimated by minimizing the weighted sum of log likelihoods of        two signals. Signal likelihood is calculated by Gaussian        probability density function.    -   Naive version of SiFu: This localization scheme uses the same        set of signals as SiFu, namely, WiFi, magnetic field, GPS and        Google to compute likelihoods, and then fuses with INS signals        using a particle filter. In the naive version, signals are not        weighted. The joint measurement likelihood is computed only by        multiplying all the likelihoods together.    -   Google Fused Location API: A fusion localization algorithm        created by Google that intelligently combines signals including        WiFi, Bluetooth and GP S. The following performance metrics are        for comparison:    -   Localization error: Localization error is the Euclidean distance        between the estimated location and the ground truth. It is the        most direct and common metric to measure the overall performance        of localization systems. Error distributions are examined for a        more comprehensive evaluation.    -   Computation time: Computation efficiency is of great importance        for localization systems, especially for real-time systems. In        this Example, the average time required to compute the location        is measured after signals have been received.

The likelihood computation module is first validated for spot-based RSSIvectors. The model used in the present invention which computes based onthe distance in the latent space is compared with the other schemes thatare based on different metrics, such as Euclidean distance and cosinesimilarity, in the raw signal readings. FIG. 5 shows the comparisonbetween different sites. The SiFu system of the present inventionperforms more accurately in all three sites. In the corridor area, thethree comparison schemes perform almost the same, probably because WiFisignals there are stronger and more stable. However, when tested in moreopen areas like the indoor open space and atrium, the multi-path fadingeffect becomes much stronger and WiFi signals may fluctuate greatly.Naive methods such as comparing the Euclidean distance and cosinesimilarity in the raw signal space are evidently affected. Instead, thesystem of the present invention has already learned to adapt the signalnoise and missing values in the training phase and is thus more robustand achieves superior accuracy over the other systems.

For fusion localization, FIG. 5 plots the mean localization error ofSiFu and the compared schemes over three test sites. SiFu demonstratesimproved accuracy over the prior art algorithms in all three sites. ForWiDeep, the performance is reasonable in the corridor area but in theindoor open space and atrium regions, its accuracy and stability suffercritically due to the noisy WiFi measurements in open areas. As WiDeeponly considers a single signal for localization, it does not performwell in regions with weak and unstable WiFi signals. In contrast, FSWfuses geomagnetism with WiFi. The negative effects of poor WiFi signalsin the open areas may be mitigated by the magnetic field, thus itgenerally works better than WiDeep. However, its fusion method is notideal as the F-score it introduces does not consider the intrinsiclocalization errors in complex environments. SiFu improves accuracy overthe prior art by fusing more signals and adopting a better fusionapproach. Fusing more signals allows the SiFu system to have greaterinformation to make decisions and further introducing INS signals andthe particle filter stabilizes the localization results. Furthermore,compared with the naive version of the present invention, SiFu takesadvantage of the weighted likelihood scheme that alleviates the impactsof bad model performance in some areas, resulting in an even lowerlocalization error. The SiFu system achieves the best mean localizationerror among the comparison schemes. The mean errors are cut by more than30% as compared to prior art systems.

FIG. 6 plots the cumulative distribution function of the localizationerrors in different test sites. It is clear that the SiFu systemachieves the most stable and robust performance among all comparedschemes. The maximum errors are reduced significantly and a largeportion of the results have acceptable localization errors in all threesites. In FIG. 6A, the performance of SiFu and its naive version isobserved to be close. This suggests that the weighted fusion method doesnot play an important role in the corridor region. This may be becauseindividual models have already worked well and the weight only haslimited effects on the shape of the likelihood function. This can alsoexplain why FSW and WiDeep can achieve reasonable accuracy in this site.FIG. 6B shows some slight impact of the weighted likelihood. The maximumerror is reduced while several more results have lower localizationerrors than the naive version. Furthermore, as shown in FIG. 6C, theimpact of the likelihood fusion method is the most significant. Whilethere are still a number of results with average accuracy using thenaive version, SiFu substantially reduces the error. In the atriumregion, the WiFi signals may not be ideal for localization as both thelocalization error and maximum error are large for WiDeep.

In FIG. 7, the real-time performance of the compared schemes is plottedfor different sites. SiFu, which is drawn using a solid line, generallyachieves stable and satisfactory performance. The error remainsconsistently low and, importantly, does not exhibit large fluctuations.Being a general localization solution, the Google Fused Location API isnot specifically engineered for the Example testing sites. Hence it isexpected that the localization performance is not as good as the othercompared schemes requiring site survey. The results also show highlevels of variation seriously which undermines user experience. FIG. 7Aindicates similar performance from the prior art methods in the corridorregion. All methods attain generally low errors with acceptablefluctuations. In indoor open space, FIG. 7B, the results are relativelyworse. The localization errors increase considerably near the end of thewalk. However, compared with other works, SiFu still achieves the mostaccurate and stable results.

Additionally, we observe a smooth transition from finger-print region tonon-fingerprint region in the atrium area as shown in FIG. 7C. Despitethe fact that the signals used for localization change, the localizationperformance is sufficiently steady such that this transition isunnoticeable. This substantiates the ability of SiFu to handle arbitrarysignal combinations.

Furthermore, the impacts of different parameters in the SiFu system onthe localization performance and computation efficiency areinvestigated. The size of the grid used for computing likelihoods isdefined as the distance between adjacent grid points. FIG. 8A, FIG. 8B,and FIG. 8C plot the mean localization errors with different grid sizesin the corridor region, indoor open space and atrium, respectively.Overall, it is found that the mean localization increases as the gridsize increases. Each reference point in the computation grid isassociated with a fingerprint signal. The likelihood computed with thefingerprint signal aims to cover the neighboring region. When the gridsize is large, the fingerprint signals may not accurately reflect thesignals that can be sampled within the neighborhood of the grid.Eventually, the likelihood computation would become incorrect. Incontrast, a denser grid contains more reference points associated withfingerprint signals. This possibly allows better signal differentiation.For example, in the particle filter, particles nearby can have distinctlikelihoods when the grid size is small, leading to a more precisedistribution.

The effect of grid size on the computation efficiency is plotted in FIG.9. A decreasing trend of computation time with increasing grid size isobserved. It is reasonable because a sparse grid means that likelihoodshave to be computed at fewer reference points, and hence lesscomputation is required. Consequently, higher accuracy with smaller gridsize comes with a trade-off, which is computation efficiency.Furthermore, the results show that the computation complexity of theSiFu system is not high, so that the system can be easily deployed as areal-time localization system. Even in larger sites, the computationtime will not grow significantly when the magnetic field matching spaceis limited and each module does not require expensive computation.

In FIG. 10, the mean localization errors in different sites withdifferent magnetic matching ranges are shown. A V-shape can be observedfrom the plot where the localization error becomes larger when the rangeis too small or too large. As the magnetic matching space is limited bythe last predicted location, which is merely an approximate of the truelast device/user location, the true current locations may not be coveredin the matching space when the range is too small. Therefore, themagnetic field at the true location will be ignored in the sequencematching procedure and the result will not be valid. On the other hand,if the range is too large, ambiguity in magnetic sequences within thematching region may exist, which makes the likelihood computationinaccurate. In this Example, the optimal range is around 9 m, which canaccommodate the localization error while limiting ambiguity in thesearch space.

FIG. 11 plots the mean computation time with different magnetic fieldmatching range. A linearly increasing trend in computation time isobserved as the matching range increases. It suggests that the DTWalgorithm for magnetic field localization in the SiFu system is ratherexpensive. When the matching space is large and there is a need to matchagainst more magnetic field fingerprints, the computation time increasesdramatically. To make the SiFu system operable in real-time, the rangeshould be limited appropriately. Considering the localization accuracy,the default is set to 9 m.

FIG. 12 plots the mean localization errors in different sites withdifferent σ_(m). A V-shape is observed. The sensitivity parameter σ_(m)for the observation model for magnetic field decides how much noise canbe tolerated. If it is too large, the likelihood cannot effectivelydiscriminate locations because dissimilar signals will still get arather high likelihood. If it is too small, the likelihood will be loweven if the sampled signal deviates a bit from the fingerprint signal.As the noise tolerance is low, signal noise cannot be handled properly.This implies that the sensitivity parameter σ_(m) cannot be too large ortoo small in order to obtain a legitimate location estimate.

Further, the impact of σ_(w) on the localization performance indifferent sites is demonstrated in FIG. 13. A similar V-shape can beobserved. This reinforces the results in the experiment for am. Indeed,σ_(m) and σ_(w) are two important parameters in the SiFu system and theresults explain the default settings of these two parameters.

Finally, the localization performance with sparse WiFi sampling isdetermined. FIG. 14 shows the mean errors of SiFu and the comparedschemes in different test sites. The results are consistent with thedense WiFi sampling results. With sparser WiFi sampling, it can beobserved that the mean errors generally increase compared to thesituation where dense WiFi signals are available. The increase of theerror of SiFu is much lower than that of the compared prior art schemes.It remains the best localization system among the compared schemes.Furthermore, it is again noted that the weighted scheme in SiFu booststhe localization performance over the naive version of SiFu.

FIGS. 15A, 15B, and 15C plot the cumulative distribution of thelocalization errors in different sites in the sparse WiFi sampling case.WiDeep that uses only WiFi performs poorly as WiFi signals are lessfrequently available for localization. FSW, despite also consideringmagnetic field, only performs slightly better than WiDeep. This isprobably because it does not work when some signals, such as WiFi inthis case, are unavailable. When the WiFi signal is unavailable, usingonly the magnetic field may not give an acceptable performance due tothe ambiguity of the magnetic field. In contrast, SiFu fuses moresignals and integrates INS signals. Device/user location can still beapproximated accurately even though some signals are missing. Similar tothe dense WiFi sampling case, a majority of the results has satisfactorylocalization error.

As used herein and not otherwise defined, the terms “substantially,”“substantial,” “approximately” and “about” are used to describe andaccount for small variations. When used in conjunction with an event orcircumstance, the terms can encompass instances in which the event orcircumstance occurs precisely as well as instances in which the event orcircumstance occurs to a close approximation.

While the present disclosure has been described and illustrated withreference to specific embodiments thereof, these descriptions andillustrations are not limiting. It should be understood by those skilledin the art that various changes may be made and equivalents may besubstituted without departing from the true spirit and scope of thepresent disclosure as defined by the appended claims. There may be otherembodiments of the present disclosure which are not specificallyillustrated. The specification and the drawings are to be regarded asillustrative rather than restrictive. Modifications may be made to adapta particular situation, material, composition of matter, method, orprocess to the objective, spirit and scope of the present disclosure.All such modifications are intended to be within the scope of the claimsappended hereto. While the methods disclosed herein have been describedwith reference to particular operations performed in a particular order,it will be understood that these operations may be combined,sub-divided, or re-ordered to form an equivalent method withoutdeparting from the teachings of the present disclosure. Accordingly,unless specifically indicated herein, the order and grouping of theoperations are not limitations.

1. A probabilistic system fusing an arbitrary combination ofheterogeneous signals for determining the location of electronicdevices, comprising: a signal sampling device for detecting one or moresignals emitted by an electronic device to be located, the one or moreemitted signals selected from one or more of geolocation signals, WiFisignals, Bluetooth signals, 4G communication signals, 5G communicationsignals, geomagnetism signals, or inertial navigation system signals(INS), the signal sampling device using heterogeneous sampling rates foreach of the selected signals; a likelihood processor cooperating withthe signal sampling device to receive information about selected sampledsignals, the likelihood processor creating a grid of reference pointsfor an interested area in which the electronic device may be located;the likelihood processor independently computing, for each selectedsampled signal, a location likelihood that is a probability of observingthe sampled signal given that the electronic device is located atdifferent reference points in the grid and fusing the locationlikelihoods for plural signals in a weighted fashion.
 2. Theprobabilistic system fusing an arbitrary combination of heterogeneoussignals for determining the location of electronic devices according toclaim 1, wherein the likelihood processor includes a likelihood fusionmodule for assigning weights to contrast the location likelihoods. 3.The probabilistic system fusing an arbitrary combination ofheterogeneous signals for determining the location of electronic devicesaccording to claim 2, further comprising a particle filter, the particlefilter using weighted location likelihoods from the fusion module toupdate a particle weight in the particle filter to determine thelocation of the electronic device.
 4. The probabilistic system fusing anarbitrary combination of heterogeneous signals for determining thelocation of electronic devices according to claim 1, wherein thelikelihood processor includes a neural network for processing RSSIvector-based signals.
 5. The probabilistic system fusing an arbitrarycombination of heterogeneous signals for determining the location ofelectronic devices according to claim 1, wherein the likelihoodprocessor includes a Bayesian analysis module for determining thelikelihood location of geolocation signals.
 6. The probabilistic systemfusing an arbitrary combination of heterogeneous signals for determiningthe location of electronic devices according to claim 1, wherein thelikelihood processor includes a dynamic time warping module for analysisof sequence-based signals.
 7. A localization method fusing an arbitrarycombination of heterogeneous signals for determining the location ofelectronic devices, comprising: heterogeneously sampling one or moresignals emitted from an electronic device to be located, the one or moresignals selected from one or more of geolocation signals, WiFi signals,Bluetooth signals, 4G communication signals, 5G communication signals,geomagnetism signals, or inertial navigation system signals (INS);feeding sampled signal information to a likelihood processor cooperatingwith the signal sampling device, the likelihood processor creating agrid of reference points for an interested area in which the electronicdevice may be located; the likelihood processor independently computing,for each selected sampled signal, a location likelihood that is aprobability of observing the sampled signal given that the electronicdevice is located at different reference points in the grid and fusingthe location likelihoods for plural sampled signals in a weightedfashion; wherein the method dynamically drops a sampled signal when thesampled signal is not observed at a localization time.
 8. Thelocalization method fusing an arbitrary combination of heterogeneoussignals for determining the location of electronic devices according toclaim 7, further comprising using a particle filter to dynamicallycombine weighted location likelihoods with inertial sensor measurements.9. The localization method fusing an arbitrary combination ofheterogeneous signals for determining the location of electronic devicesaccording to claim 7, wherein the likelihood processor includes alikelihood fusion module for assigning weights to contrast the locationlikelihoods.
 10. The localization method fusing an arbitrary combinationof heterogeneous signals for determining the location of electronicdevices according to claim 7, wherein the likelihood processor includesa neural network for processing RSSI vector-based signals.
 11. Thelocalization method fusing an arbitrary combination of heterogeneoussignals for determining the location of electronic devices according toclaim 7, wherein the likelihood processor includes a Bayesian analysismodule for determining the likelihood location of geolocation signals.12. The localization method fusing an arbitrary combination ofheterogeneous signals for determining the location of electronic devicesaccording to claim 7, wherein the likelihood processor includes adynamic time warping module for analysis of sequence-based signals.